The invention relates generally to manipulating image information and more particularly to methods and systems for reconstructing an image from a mosaic pattern of intensity values for different colors.
A digital color image consists of an array of pixel values representing the color intensities at each point of the image. Typically, three colors are used to generate the image, with the intensity of each of the colors being identified for each pixel. In reconstructing the image, the three intensities are combined on a pixel-to-pixel basis to determine the image that is printed or displayed to a viewer.
Conventional color photography utilizes three overlapping color sensing layers having sensitivities in different regions of the color spectrum. Typically, red, green and blue are the three selected colors. However, digital cameras are limited to a single array of sensors, so that there is only one xe2x80x9clayer.xe2x80x9d Within this layer, each sensor determines the intensity of a particular color. As a result, the sensor array does not produce a color image in the traditional sense, but rather a collection of individual color samples. The assignment of the colors to the individual pixels within the array is sometimes referred to as the color filter array (CFA) or the color xe2x80x9cmosaic pattern.xe2x80x9d To produce a true color image, with a full set of color samples at each pixel, a substantial amount of computation is required to estimate the missing information. The computational operation is typically referred to as xe2x80x9cdemosaicing.xe2x80x9d
Since only one color is sensed at each pixel, two-thirds of the spectral information is missing. To generate the missing information at a particular pixel, information from neighboring pixels may be used, because there is a statistical dependency among the pixels in the same region of a particular image. A number of demosaicing algorithms are available. xe2x80x9cBilinear interpolationxe2x80x9d is an approach that attempts to minimize the complexity of the demosaicing operation. A bilinear interpolation algorithm interpolates color intensities only from the same-color sensors. That is, the red sensors are treated independently from the green and blue sensors, the green sensors are treated independently from the red and blue sensors, and the blue sensors are treated independently from the red and green sensors. To provide a red intensity value at a given pixel, the values measured by the red sensors in a designated size neighborhood (e.g., in a neighborhood of nine sensors having the given pixel as its center) are interpolated. If the mosaic pattern of sensors is a Bayer pattern (i.e., a repeating 2xc3x972 sensor array kernel having two green sensors diagonally positioned within the kernel and having one red sensor and one blue sensor), the bilinear interpolation algorithm may use twelve kernels of convolution to reconstruct the image. However, while the approach of isolating the different colors in the demosaicing operation provides a relatively low level of computational overhead, the reconstructed image lacks the sharpness of conventional color photography. As will be explained below, there is an unfortunate tradeoff between minimizing artifacts and maximizing sharpness.
Another approach is described in U.S. Pat. No. 5,475,769 to Wober et al. In this approach, the color sensors are no longer treated as being independent of sensors of other colors. As in the bilinear interpolation approach, a defined-size neighborhood of pixels is xe2x80x9cmovedxe2x80x9d about the mosaic pattern and an intensity value is determined for the pixel that is at the center of the neighborhood. Each pixel in the neighborhood is weighted for each color sensitivity relative to the center pixel. The missing color intensity values for each sensor pixel are estimated by averaging the weighted pixels in the neighborhood for each color band. The neighborhood data and the pixel data are arranged in a matrix and vector expression having the form A*W=X, where A is a matrix of neighborhood values, W is a vector representing the weighting coefficients, and X is a vector of color components of the particular wavelength of the center pixel. The weighting coefficients may be determined using a linear minimum mean square error (LMMSE) solution.
A single pixel array may be viewed as consisting of a number of separate planes of pixels in which each plane has sensors of the same color. Since the pixels do not overlap, the sensors in the various planes are at different locations. Demosaicing approaches that take weighted averages across more than one monochromatic plane make use of the statistical dependencies among the sample locations. In effect, the blurring of an image by the camera optics allows a true object edge (i.e., an edge of one of the objects being imaged) that would exist precisely on the sensors of one of the monochromatic planes to also be seen in the other color planes, since the image is spread by the blurring onto the sensors of the other monochromatic planes.
While the blurring that is introduced by the camera optics aids in the demosaicing operation to reconstruct an image, the demosaicing operation should not enhance the blurring or add any other artifacts to the reconstructed image. The goal of demosaicing algorithms should be to faithfully reconstruct the image, so as to provide a true-color, sharp print or display. One problem is that the interpolation approaches tend to introduce artifacts at abrupt changes in intensity. As an example of an abrupt transition, a digital picture that is captured in a poorly lit room will have a sharp transition along any source of illumination. Sharp transitions also occur at borders between two bright colors. Because most known interpolation algorithms provide estimation within a neighborhood that is not dependent upon identifying transitions, the averaging that occurs along the transitions will consider both the high intensity values on one side of the transition and the low intensity values on the opposite side. This space-invariant interpolation dulls the image along the transition. That is, the reconstructed image loses some of its sharpness. The degree of sharpness loss will partially depend on the emphasis on controlling the introduction of artifacts at the abrupt intensity transitions (i.e., there is a sharpness/artifact tradeoff).
A space-invariant approach that is designed to increase sharpness may be referred to as the Generalized Image Demosaicing and Enhancement (GIDE) approach of David Taubman. GIDE minimizes an error estimate of the original scene reconstruction on the basis of factors that include sensor characteristics, models of the relevant imaging system and Wide Sense Stationary (WSS) noise, and prior image statistical models. The approach is outlined in a publication by David Taubman entitled, xe2x80x9cGeneralized Wiener Reconstruction of Images from Colour Sensor Data Using a Scale Invariant Prior,xe2x80x9d Proceedings of the International Conference on Image Processing (ICIP 2000), Vancouver, Canada, September, 2000. While the approach works well in most instances, there is a tendency to xe2x80x9coversharpenxe2x80x9d at the edges of abrupt changes in intensity. That is, if there is an abrupt transition from a first intensity level to a significantly higher second intensity level, the interpolated pixel values for pixels immediately before the transition may drop significantly below the first level, while the pixels located at and immediately after the abrupt transition may be assigned interpolated pixel values that are significantly above the second intensity level. The over-sharpening introduces artifacts into the reconstructed image. While the artifacts are less visually displeasurable than those introduced by other approaches, the artifacts are apparent.
A space-variant approach is described in U.S. Pat. No. 5,373,322 to Laroche et al. and U.S. Pat. No. 5,382,976 to Hibbard, each of which is assigned to Eastman Kodak Company. The Laroche et al. patent relates to interpolating chrominance values, while the Hibbard patent uses the same techniques for interpolating luminance values. Regarding Laroche et al., a digital processor obtains gradient values from the differences between chrominance values in the vertical image directions and the horizontal image directions from a particular pixel of interest. The gradient values are compared with each other in order to select one of the directions as the preferred orientation for the interpolation of additional chrominance values. The interpolation is then performed upon values that are selected so as to agree with the preferred orientation. By performing the comparison of the gradient values, edge artifacts are reduced and image sharpness is enhanced.
U.S. Pat. No. 5,875,040 to Matraszek et al. is another Eastman Kodak Company patent that calculates a direction before pixel interpolation is performed. The Matraszek et al. patent is used to correct defect pixels in a defect region of a source image. The method includes the step of searching for edges in a non-defect region of the image having the defect region. Once the edges of the non-defect region are identified, pixel signals for the defect region are based on the pixel signals for non-defect pixels lying in a direction from the defect pixel which is determined by the direction of the edge. Preferably, both a directional angle and a magnitude are calculated in the detection of the edges. An edge directional angle is defined as being perpendicular to the gradient direction at the pixel being considered. One way of locating the edges is described as selecting the non-defect region as a xe2x80x9cslidingxe2x80x9d window of limited dimensions adjacent to the defect region, with the window defining a local non-defect region. The patent notes that adjacent does not necessarily mean immediately adjacent, since there may be some preselected distance separating the center of the window or the end of the window from the defect region. The edge of the window can then be located using a gradient field. Using the described method, pixel values can be based upon pixel values from non-defect regions.
By properly selecting the direction from which other pixel values are to be selected in the determination of a particular pixel value, the introduction of artifacts can be controlled. However, the process of first determining the direction and then determining the pixel value is computationally complex. Numerous iterations may be necessary.
What is needed is a method and system for faithfully reconstructing an image from a mosaic pattern of color information without a high level of computational overhead.
Preservation of intensity transitions is enhanced in a demosaicing operation by utilizing bilateral filtering, which will be explained below. As a result, interpolations among neighboring pixels in a mosaic pattern of color pixels are based upon factors that include the relative position and photometric similarity of pixel values within a window that is intensity insensitive. By xe2x80x9cintensity insensitivexe2x80x9d what is meant is that the position of the window is not a function of photometric similarity or dissimilarity within an image. Rather, photometric similarity is a factor after the position of the window has been determined. Thus, within a fixed-orientation window for determining an intensity value for a specific pixel, intensity values of pixels that are physically close are given greater weight than intensity values of more distant pixels and, simultaneously, intensity values that are quantitatively similar are given greater weight than intensity values that are quantitatively dissimilar. By reducing the effect of pixels on one side of an abrupt intensity transition in determining the interpolated intensity values on the opposite side of the abrupt intensity transition, the sharpness of the transition is enhanced in the reconstructed image. Thus, the adverse effects of the conventional tradeoff between sharpness and artifact-free reconstruction are at least partially eliminated.
While the invention may be used with bilinear interpolation demosaicing, another embodiment is one in which all of the intensity values within a given-sized window of the mosaic pattern are considered in determining the intensity values for pixels in the general center of the neighborhood (xe2x80x9cgeneralxe2x80x9d center because the precise center may not coincide with the physical location of any pixel, depending on the parameters of the given-sized window). Thus, demosaicing includes utilizing at least Psxc3x97Pt different convolution kernels, where Ps is the number of possible color arrangements within the given-sized neighborhood and where Pt is the number of different colors In the mosaic pattern. For a Bayer pattern Ps=4 and Pt=3, so that there will be twelve convolution kernels. From the previous description of the Bayer pattern, it can be recognized that a small scale implementation of this embodiment, from which larger scale mosaic patterns may be extended, would be
In the bilateral filtering, a similarity function that is indicative of the photometric similarity among intensity values of neighboring pixels is algorithmically combined with determinations of the relative position of the neighboring pixel locations. For each pixel in an image to be reconstructed, the similarity function is applied after the neighborhood of relevant pixels has been selected. As a result of the algorithmic combination, statistical dependencies among the intensity values within the selected neighborhood are more reliably estimated. In one embodiment, the similarity function defines coefficients that are multiplied by the determinations of relative position. In this embodiment, the similarity function returns coefficients that are close to xe2x80x9c1xe2x80x9d for small absolute photometric dissimilarity and tends to xe2x80x9c0xe2x80x9d as the photometric dissimilarity is increased. Thus, the similarity function will have little effect in image regions with slowly varying intensities, but will inhibit artifacts around highly contrasted transitions within the image.
The demosaicing process may be considered as forming a number of monochromatic image planes, such as a green image plane, a red image plane and a blue image plane. In one embodiment, each output image plane has the same number of pixels as the original mosaic pattern and each pixel includes an intensity value. Thus, each pixel location is assigned three intensity values (i.e., one intensity value for each monochromatic image), with each intensity value being based upon the relative positions of the pixel location to selected other pixel locations and photometric similarities to intensity values of the selected pixel locations. The selected pixel locations are those that are contained within the given-sized window that has the particular pixel location as its general center, since the position and orientation of the window are intensity insensitive. It should be noted that the shape of the window is not important to the practice of the invention.
The system for carrying out the invention includes a two-dimensional array of sensors in which the sensors are color-specific. While not critical, the sensors may be arranged in the Bayer pattern. Connected to the sensor array is an input that receives signals in a format representative of the mosaic pattern. Demosaicing process software/hardware is enabled to generate intensity values for each of the different colors at each of the discrete pixel locations of the original mosaic pattern. The demosaicing capability includes a bilateral filter that is configured to access the spectral information of the mosaic pattern and to process the spectral information such that intensity values are generated on the basis of both relative position of pixel locations and photometric similarity of spectral information. In one embodiment, the demosaicing capability also includes a multiplier having a first input that is responsive to calculations of relative position between first and second pixel locations within the mosaic pattern and having a second input that is responsive to calculations of photometric similarity between the spectral information contained within the first and second pixel locations. For embodiments in which the mosaic pattern is the Bayer pattern, twelve stored convolution kernels having adaptive coefficients are utilized.
An advantage of the invention is that known linear reconstruction filters, such as Wiener filters, may be modified to become transition and texture sensitive. Apart from being adaptive, the demosaicing having bilateral filtering requires no iterations. This is in contrast to systems in which edges of abrupt intensity (color or luminance) must be identified before interpolation can occur. The transition and texture sensitivity is achieved by causing the process to become space variant, but with only a small amount of additional computational overhead. Applying bilateral filtering to xe2x80x9ccorrectxe2x80x9d artifacts in demosaicing may be used in applications such as digital cameras in order to enhance sharpness in the reconstruction of a substantially artifact-free image.